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ABSTRACT 



The validation of a classroom observation instrument 



to be used in evaluating teacher interns under the Kentucky Teacher 
Intern Program is described. The validation project began in 1988. 
Fourteen educators from state universities comprised a jury that 
judged the initial version of the instrument and compared it with the 
instrument previously used, an observation instrument from the 
Florida Performance Measurement System. Participants applied the 
instrument to videotaped samples of classroom instruction. For 
determination of test-retest reliability, subjects from the 
validation phase oriented 28 local educators with regard to the use 
of the revised instrument. Each participant coded two videotapes and 
one actual intern in the classroom. This test established that the 
instrument had sufficient reliability and validity to be used in the 
KentucJcy program. Data indicate the need for a small group of 
knowledgeable people to revise the coding manual and make some 
revisions to the coding instrun^ Thirteen tables are included. 
Five appendices present the original and revised Kentucky Teacher 
Internship Program Classroom Observation instruments and supplemental 
information about the development of the instruments, (SLD) 
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The Evaluation of Intern Teachers: 

The Development of an Observation Instrument 



INTRODUCTION 

When it began, the Kentucky Teacher Intern Program (KTIP) used 
materials developed as part of the Florida Performance Measurement System 
(FPMS); however, modifications prompted the Kentucky Department of Education to 
develop its own materials to use in observing the classroom behaviors of Kentucky's 
beginning teachers. Hie evolution of a specific Kentucky instrument b?gan in 1984 
when the General Assembly enacted legislation which establishf^d tb.e Kentucky 
Career Ladder Commission. That Commission did a considerable amount of develop- 
mental work in constructing a classroom observation instrument on which student 
behaviors could be coded and summarized. 

In its final report, the Commission documents the steps involved in the 
development of a coding instrument and coding manual to determine if a standard 
format could be used to observe a classroom teacher and later used to determine if the 
teacher should receive a rating of satisfactory or outstanding. The classroom 
observation instrument was one part of a multi pronged process used to determine if 
the teacher would receive a monetary rewanl for outstanding work. 

In the process of developing the observation instrument, the Career Ladder 
Commission reviewed national efforts to help and support first year teachers. The 
process included reviewing various induction models being used nationwide. The 
FPMS coding instrument appears in Peterson, Kromrey, Micceri, and Smith (1987), 
and the Kentucky Career Ladder instrument appears in the Commission's Final 
Report (1988). 

The Advisory Committee for Teacher Improvement (ACTIP), an advisory 
group attached to the Division of Teacher Education and Certification, had to find a 
new instrument for collecting cla^room data and wanted to begin where the Career 
Ladder Commission ended. ACTIP asked the Division of Research to determine the 
reliability and validity of this instrument (Appendix A). 

The entire validation project began in the spnng of 1988, and the instru- 
ment was to be ready for field use by May, 1989. By inference, items could be 
removed from the KTIP because of data resulting from the project, but no items could 
be added because data would not be available to justify inclusion of untested items. 
The product also had to be more *user friendly' than the FPMS classroom observation 
instrument. 



PROJECT RATIONALK 



The Division of Research developed tlie research methodology, collected 
data, and analyzed both. The Division of Teacher Education and Certification 
scheduled regional meetings and paid participant expenses. The initial research 
project was to be completed to allow the instrument to be used in the elementary and 
secondary schools during the 1989-90 school year. The research project had to 
accomplish the following major tasks: 

1 . Establish validity for the Kentucky instrument. 

2. Determine the instrument's utility and its raters* ability to use it 
accurately and consistently. 

3. Collect data which would aid in producing a revised Kentucky instru- 
ment and coding manual. 

The project staff utilized two pre-experimental research designs (Stanley 
and Campbell 1963} to accomplish the three tasks. The entire project would create 
four separate data bases while using six training video tapes - two elementary, two 
middle school, and two high school tapes - reviewed by various subjects at different 
times and coded using the FPMS or/and KTIP instrument. 

The content validity of the newly developed KTIP classroom observation 
instrument was effectuated by employing a process similar to that expressed by 
Kerlinger (1987). The ACTIP had to develop an instrument that would be easier for 
observers to code than the FPMS, yet its form and content had to be more accessible 
and adequately represent observable teacher behaviors. 

Determining the reliability of the KTIP instrument required anchoring it 
to another instrument that had some record of reliability and validity, and the FPMS 
was the obvious choice. A jury of experts inspected the KTIP instrument and 
determined which items therein were exact or partial representations of items on the 
FPMS instrument. Those six people represented staff members from the Divisions of 
Research and Teacher Education and Certification and University staff members 
who had been involved in the Career Ladder Commission work and the teacher intern 
program. The crosswalk emphasized those teacher behaviors which were coded by 
making hash marks. Behaviors reflected in items on the back of the KTIP and on the 
cover of the FPMS document indicated with a check mark were not included in this 
analysis. The six crosswalkers represented the spectrum of potential jurists. No 
ovtrt selection process restricted the variation of the six raters. 



INSTRUMENT DEVELOPMENT 

Tables lA through 10 array the jury of experts' consensus markings as 
determined by the Division of Research staff. KTEP category refers to subparts of 
seven sequential sections inside the coding instrument from Content % **States/ 



defines content focus" through Time 1 "Mismanages instructional time**. The FPMS 
category indicates items from that instrument which crosswalkers matched to the 
KTIP category listed on the left. For example, crosswalker #1 matched 3L **Orients 
students to classwork/maintains academic focus" on FPMS to Content 1 "States/ 
defines content focus**. Crosswalker #2 identified IIL Treats conceptrdefinition/ 
attributes/examples/non-examples** as well as 13L "States and applies academic rule** 
as partial or complete matches. Since FPMS utilizes left (L) and right (R) markings, 
an algorithm was built which utilized certain L behaviors in some cases and some R 
behaviors in other cases. 



Table lA. displays the jury's findings in the content section. This group 
matched all seven of the content teaching behaviors with items on the FPMS. 



TABLKIA* 

Cross- Reference Matrix 
KTIP Classroom Ob^rvalion Instrument (Content Section) 
Florida Performance Measurement System 




Kilt L«ategory 




r flflo V/ftiegory 




StateH/defines content focus 


(CD 


Orients students to classwork/ 
maintains academic focus 

Treats conceptdefinition/attributes/ 
examples/nonexamples 


(3L) 
(IIT) 


Kxplains content clearly 


{C2) 


Treats concept definition/attributes 
examples/non-examples 

Stales and applies academic rule 


{5AR) 
{13L) 


Models/applies content focus 


(03) 


Treats concept-definition/attributes 
examp!es/non-examples 

Discusses cause-efTect/uses linking 
words/applies law or principle 


(llU 
{12L) 


Checks for comprehension 


(C4) 


Gives dircctiona/assigns/checks 
comprehension of homework, seatwork 
assignment/gives feedback 

Questions: academic comprehension/ 
lesson development 


(9L) 
(5A,BU 


Provides guided practice 


(C5) 


Provides for practice 
Circulates and assists students 


(BL) 
(lOL) 


Conducts review/summary 


(C6) 


Conducts beginning/ending review 


{4U 


Provides independent practice 


(07) 


Gives dirMtions/assigns/checks 
comprehension of homework, seatwork 
assignment^gives fe^back 


(9L) 
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Table IB. deals with the use of questioning techniques. Of the nine items 
on the KTTP, the experts found five matches. 



TABLE IB. 

Cross- Reference Matrix 
KTIP Classroom Observation Instrument (Questioning Techniques Section) 
Florida Performance Measurement System 



KTIP Category 




FPMS Category 




Asks academic question 


(QTl) 


Questions: academic comprehension/ 




lesson development 


(5A,BU 


Asks multiple question 


(QT2) 


Poi^s multiple questions asked as 


{5AR) 






one.unison response 


Asks/i»uses/names 


(QT3) 






Does not provide wait 


{QT4) 






Guides reciter 


(QT5) 






Does not guide reciter 


(QT6) 






Allows call-outs 


(QT7) 


Poses multiple questions asked as 


(5AR) 






one, unison res{»nse 


Asks procedural questions 


(QT8) 


Poses non-academic questions/ 






nonacademic procedural questions 


(5BR) 


Asks unrelated questions 


{QT9) 


Poses non-acadomic questions/ 






nonacademic procedural questions 


{5BR) 



Table IC. analyzes the section detailing the intern's reac-ions to student 
responses. In three of six response situations, the jury found similarity. 





TABLE IC. 




Cross- Reference Matrix 
KTIP Classroom Observation Instrument (Responses Section) 
Florida Performance Measurement System 




KTIP Category 




FPMS Category 




Acknowledges student 
responses 


(Rl) 


Recognizes response/amplifies/gives 
corrective feedoack 


(6U 


Rephrases/amplifies student 
responses 


(R2) 


Recognizes response/amplifies/gives 
corrective feedback 


(6L) 


Corrects/clarifies student 
responses 


{R3) 


Recognizes response/ampliTies/gives 
corrective feedback 


(6L) 


l>oes not correct/clarify 


(R4) 






responses 








Responds to academic 
questions/input 


(R5) 






Responds u> non-academic 
questions/input 


(R6) 







TaMe ID. depicts the manner in which both instruments allow the 
reviewer to record praise. The group established matches in three of four praise 
areas. 



TABLE ID. 

Cross- Reference Matrix 
KTIP Classroom Observation instrument (Praise Section) 
Florida Performance Measurement System 



KTIP Category 




FPMS Category 




Uses specific academic praise 


CPl) 


Gives specific academic praise 


(71,) 


Uses genera! academic praise 


{P2) 


Uses general, non-specific praise 


(7R) 


Uses group academic praise 


(P3) 


Uses general, non-specific praise 


(7R) 


Uses conduct-related praise 


{H4) 







Table 1£. concerns student motivation and is labeled communication* The 
cross walkers discerned parallels two of four times. 



TABLE IK. 

Cross- Reference Matrix 
KTJF Classroom Observation Instrument (Communication Section) 
Florida Performance Measurement System 



KTIP Category KPMS Category 

Cues students (COD 



V ses emphasis (C02) 
Uses challenge (C03) 

Uses sarcasm/negative effect {C04) 



Kxpresscs enthusiasm/verbally 

chal lenges students ( 1 6L) 

Ignores student or i^sponse/ex presses 
sarcasm, disgust, harshness (6R) 



Table IF. (»mpares methods of dealing with student oiisconduct. 
detected comparisons for all four KTIP items. 



The jury 



TABLE IF. 

Cross-Referenee Matrix 
KTIP Classroom Obi$ervatxon Instrument (Management Section) 
Florida Performance Measurement Svntem 



KTIP Category 

Stops misconduct positively (Ml ) 

Stops misconduct negatively (M2) 

l>>es not stop misconduct {M3) 

Uses sarcasm/negative effect (M4) 



FPMS Category 

Stops misconduct (20L) 
Delays dcsisL^docsn't stop 

misconducl/desisU punitively (20K) 
Delays desist/doesn't stop 

miscondurL/desists punitively (20K) 
Maintains instructional momentum (21 L) 



Table IG. shows how the two instnmients deal with appropriate use of time 
in the classroom. Again both items had nimierous matches on the FPMS. 



TABLE la 

Cross- Reference Matrix 
KTIP Classroom Observation Instrument (Time Section) 
Florida Performance Measurement System 




KTIP Category 


FPMS Category 




Minimizes management time 


(Tl) Begins instruction promptly 


(ID 




Handles materials in an orderly manner 


(2L) 




Maintains instructional momentum 


(2 ID 


Manages instructional time 


(T2) Delays 


(IR) 




Does not organise or handle materials 
systematically 


(2R) 




Loses momentum - fragments 
non-academic directions, overdwells 


(21 R) 



A review of all seven subsections of Table 1. establishes a consensus from 
the jury of experts that 26 of the 36 KTIP teaching behaviors are recordable on the 
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FPMS. Both forms contained extraneous items such as the physical features of the 
classroom and years of teaching experience. Because many of the matching items 
were like these and not considered, the similarity depicted between the two instru- 
ments in Table 1. is understated. 

During the period of time from May 31, 1988 through June 3, 1988 two 
representatives from each of the eight state universities met in Bowling Green, 
Kentucky to participate in process development and data collection. Fourteen of the 
16 attendees made up the participant pool. On the first day the participants viewed 
the six tapes from the Florida system and coded on FPMS instruments in accordance 
with FPMS standards and practices. Through the next three days the fourteen 
participants were trained to use the KTIP instrument and discussed detailed differ- 
ences between FPMS and KTIP instruments. Prior to closing the session on Friday, 
each participant again viewed the same six FPMS video tapes. This time the parti- 
cipants marked the behaviors viewed on the KTIP instrimient The data gleaned 
from these codings became two of the four data bases. 

The Division of Research examined the degree of correlation between 
expert responses on the 26 common items and found it to be significant. Summarized 
in TABLE 2, the analysis compares both codings on the six video tapes by each of the 
14 subjects. Each subject had a total of 156 paired scores. Those paired scores were 
used to calculate the correlation coefficient for each subject Based on video tapes 
which were samples of instruction, the range of markings was rather narrow. The 
correlation coefficients were significant beyond the .001 level for each of the fourteen 
subjects. When the correlations were examined by video tape for each subject, 79 of 
the 84 analyses resulted in statistically significant correlations at the .05 level. 





TABLE 2. 


Concurrent Validity (Pearson Product- Moment Correlation Cocfncients) 




KTIP and FPMS ratings 


Using Table lA-lG Cross- iieference 


Subject Number r for Six Tapes Combines 


1045 


0.6198 


1264 


0.6191 


1380 


0.5858 


2581 


0.5864 


3527 


0.5786 


4560 


0.6739 


5101 


0.5340 


5366 


0.4670 


6201 


0.6133 


6268 


0.6332 


6788 


0.5892 


7188 


0.6550 


8810 


0.6784 


9333 


0.5349 




p<,OOJ for all correlations 
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Table 3. summarizes the individual correlation coefficients for each subject 
for each video tape. Only five of 84 comparisons produced nonsignificant results. 



TABLK3. 

Concurrent Validity (PettrK)n Product- Moment Correlation Coefficients) 

KTl P and KPMS ratings 
Using Tabic lA 10 Cross-reference 





TAPK 


TAPE 


TAI»E 


TAPE 


TAPE 


TAPE 




I 


2 


3 


4 


S 


6 


1045 


0.76 


0.74 


0.b5 


0.47 


0.70 


0.70 


1264 


066 


0.71 


0.72 


0.52 


0.61 


0.74 


1380 


0.56 


071 


0.75 


0.55 


0.62 


0.68 


2581 


0 54 


0 71 


0.70 


0.48 


0.50 


0.70 


3527 


0.59 


0.62 


0.51 


0.46 


0.34» 


0.70 


4560 


0.49 


0.71 


0.72 


0.35* 


0.68 


0.76 


5101 


0.54 


0.65 


0.54 


0.39 


033» 


0.59 


5366 


0.48 


0.56 


0.71 


0.14* 


0.67 


0.67 


6201 


0.41 


0.69 


0.49 


0.48 


0.49 


0.62 


6268 


0.69 


0.68 


0.55 


0,55 


0.68 


0.68 


6788 


0.66 


0.71 


0.62 


0.46 


0.58 


0.68 


7188 


0.69 


0.74 


0.69 


0.54 


0.58 


0.70 


8810 


0.55 


0.73 


0.71 


0.59 


0.68 


0.70 


9333 


0.54 


0.71 


0.58 


0.32» 


0.57 


0.65 



* nomign'ficant result 



PILOT TEST 

The second task, determination of test'retest reliability, used the subjects 
from the validation phase to orient other subjects in the use of the KTIP classroom 
observation instnmient. No contn)! or comparison group existed to contrast with the 
oriented groups. The design called for the trainers representing the state uni- 
versities to select from their service area 28 local educators to receive two day 
orientation sessions. Each of the seven regional groups was to contain two partici- 
pants from the following classifications. The result would be 28 participants in each 
regior elementary principal, middle school principal, high school principal, voca- 
tional educators, college elementary specialists, college middle school specialists, 
college high school specialists, elementary resour *e teachers, middle school resource 
teachers, high school resource teachers, elementary instructional supervisors, high 
school instructional supervisors, exceptional educators ( elementary, secondary or 
college). 

Persons besides those listed were welcome to attend the orientation 
sessions. Not all sessions could include all of the 28 people specified. Each parti- 
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cipant completing all phases received a stipend for services rendered. Orientation 
sessions were conducted in seven regions; as a cost saving measure one region did not 
participate in the reliability determination phase. Appendix C tables group charac- 
teristics. Each participant had to code two tapes as part of the orientation. Each 
person then had ninety days to code an actual intern in the classroom using the KTIP 
instrument The participants were also instructed to make any written comments 
they deemed appropriate about the instrument or coding manual. The comments 
would be turned in when the participants returned to the orientation site again to 
code two tapes, which would become the retest data set. 

The seven orientation sessions were held between late July, 1988 and 
August, 1988. Each orientation session had the same agenda and each was conducted 
by local staff members. The morning portion called for reviewing the KTIP coding 
manual, coding practice video tapes, and other activities to get the local educators 
acclimated to the KTIP Instnunent. In the afternoon portion, the ^^up was divided 
by level (elementary, middle, or secondary). Each group viewed two of the six video 
tapes designated for its level and used the KTIP instrument to code. This design 
allowed the data to be sorted by geographical area, by level, and by responsibility and 
served as a pretest to determine test-retest reliability while building data base three. 



DATA ANALYSIS 

The instrument yielded three different data subsets for each category. The 
three subsets were: (1) content determination, (2) hashmark section, and (3) check- 
mark section. The content area presented the largest problem in quantification 
because each coding participant saw different content being taught. Some people 
viewed a tape and saw many different content areas; others saw a single content 
being taught. Later discussions revealed that these differences resulted from the 
background of participants in specific curriculum areas. Subjects who were knowl- 
edgeable in a certain area tended to indicate a larger number of components than an 
individual who was not as highly versed in that specialty area. The participant was 
required to list the content in the slant lines above the content descriptors and place a 
check mark after each of the seven content items if/as the teacher on the video tape 
completed each. The data set resulted in a variety of marking patterns for the same 
tape. One should also note that the concept of 'content* checkmarks and wiiting the 
content area was new to some of the participants. Although a similar procedure was 
used in the Career Ladder project, only a small sample of the participants in this 
project had used the procedure. 

Since the sessions were for orientation purposes, the participants did not 
get a great amount of time to practice coding this section. One participant could have 
marked one content area while another marked four content areas for the same tape. 
Table 4 arra}^ a summary of pre- and post markings. With many possible markings, 
a rigorous statistical analysis was not appropriate. The table entries report the 
resultant distribution of checkmarks as being either unimodal or multimodal. In the 



ERIC 



-9- 



pretest distributions, tapes 1, 5, and 6 were marked as either one or two modes, not 
mixed, while tapes 2, 3, and 4 were mixed. The number of subjects marking eacL tape 
is reported beside the '^N'*. A large difference in the participant count resulted 
because one orientation site did not collect data in the same design as the other six 
sites. This inconsistency caused the total participants responding to tapes 3 and 4 to 
be 47 end 42 on the pretest. Eight reversals in modality occurred from the pretest to 
the posttest which could have b^en the result of numerous intervening events such 
as: field practice; question and answer sessions held before the posttest data was 
collected; or exchanges between various participants as part of working as members 
of an intern committee. No effort was made to keep participants (n^m communicating 
or doing other educational activities which might influence tlie ''i.arking patterns. 
Consistency existed from the pretest to the posttest on tapes 1, 5, and 6; however, one 
reversal was noted on tape 4; two reversals on tape 2; and five reversals on tape 4. 
Overall, 80.9% or 34 out of 42 categories retained their original modality. Tape 2 was 
an elementary tape and tapes 3 and 4 were middle school tapes. 

A chi square analysis performed on the two data matrices used the pre- 
matrix as the expected outcome for the post matrix. In the prematrix 27 cells were 
one mode and 15 cells were bimodal cells resulting in a chi square value of .30. No 
significant difference occurred when comparing the cells of the posttest to the cells jf 
the pretest. 







TABLK4. 














Cross- Reference Matrix 










Modality of Markings, Pre- and Posttest, S'u Tapes 










Seven Content Areas 












TAPK 


TAPK 


T\PK 


TAPE TA"K 


TAPK 






1 


2 


3 


4 5 


6 






Vrv Fust 


Prt» Ptml 




Vrv Piist Pre P<«t 


Pre 


P«»t 




(Cl> 


ONK ONK 


ONK TWO 


ONK ONK 


ONK ONK TWO TWO 


ONK 


ONK 


Kxpiii}n»ci>nt«*nl ciparly 


«C2> 


ONK on;*: 


TWO TWO 


ONK TWO 


ONK ONK rWO TWO 


ONK 


ONK 




(C3) 


ONK ONK 


mO TWO 


ONK TWO 


TWO TWO TWO TWO 


ONK 


ONK 


Cht^cks U.r irum»*feht*':aion 


(C4) 


ONK ONK 


TWO TWO 


TWO ONK 


ONK ONK TWO TWO 


ONK 


ONK 


Provides K'jiJfd practice 




ONK ONK 


ONK TWO 


ONK TWO 


TWO ONK TWO TWO 


ONE 


ONE 


i%>nducl« revif w^ummary 


trB) 


ONK ONK 


TWO TWO 


ONK nsK 


ONK ONK TWO TWO 


ONK 


ONK 


Provides indvpvndfnt praclicv iCl ) 


ONK ONK 


ONE ONK 


TWO ONK 


ONK ONK TWO TWO 


ONK 


ONE 




N = 


70 66 


69 66 


47 4! 


42 38 67 64 


67 


64 



Table 5. depicts the hashmark sections from Questioning Technique 1 
through Time 2. The table represents the number of times subjects marked the cate- 
gories wliilc viewing the six tapes during both marking sessions. The "TOT IN** 
colunm shows the number of subjects who marked within the 70% range established 
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by the fourteen experts during the first meeting. The data indicate that 72.29% of the 
20,329 grids marked fell within acceptable standards with the percentages being 
slightly higher for pretest (75.27%) than for posttest (73.25%). The division of 
research made no elTort to validate additionally the criteria established during the 
May-June meeting. The pretest/posttest correlation was .97 for the items. 



TABLK5. 

Number and Percentage of Subjects Marking Within the Criterion 
of 70 Percent by Category for All Six Tapes 
Pre- and Post test 



CATKUORY 




WT 


ror 


roT 








IN 


Ol'T 


TOT 


IN 


Asks academic question 


(QTl) 


472 


229 


701 


67 33 


Asks mulliple questions 




405 


296 


701 


57.77 


Asks/pauses/names 


{QT3) 


406 


295 


701 


57.92 


Does not provide wait time 


(Q'r4) 


623 


78 


701 


88.87 


Guides reciter 




584 


117 


701 


83.31 


Does not guide reciter 


(QT6) 


701 


0 


701 


100.00 


Allows call -outs 


(Q'H) 


400 


301 


701 


57.06 


Asks procedural question 


(QT8) 


368 


333 


701 


52.50 


Asks unrelated question 


(QT9) 


623 


78 


701 


88.87 


Acknowledges student responses 


(Rl) 


293 


408 


701 


41.80 


Kephrases/ampiines student 












fi TV) n tt^ 

I v9{>AJXl9vo 












Corrects/clarifies student 












responses 


{K3) 


538 


163 


701 


76.75 


Does not correct/clarify 












responses 


{H4) 


694 


7 


701 


99.00 


Responds to academic 












questions/input 


{R5) 


535 


166 


701 


76.32 


Responds to non-acaden^ic 












questions/Input 


(KB) 


558 


143 


701 


79.60 


Uses specific academic praise 


iVl) 


535 


166 


701 


76.32 


Uses general academic praise 


(P2) 


423 


278 


701 


60,34 


Uses group academic praise 


(P3) 


650 


51 


701 


92.72 


Uses conduct-related praise 


(H4) 


631 


70 


701 


90.01 


Cues students 


(COD 


167 


534 


701 


23.82 


Uses emphasis 


(C02) 


501 


200 


701 


71.47 


Uses challenge 


{C03) 


634 


67 


701 


90.44 


Uses sarcasm/negative affect 


(C04) 


634 


67 


701 


90.44 


Stops conduct positively 


(Ml) 


554 


147 


701 


79.03 


Stops conduct negatively 


(M2) 


643 


58 


701 


91.73 


Does not stop misconduct 


(M3; 


656 


45 


701 


93.58 


Manag^'s overlapping events 


(M4) 


612 


89 


701 


87.3? 


Minimizes mana^rinent time 


(Tl) 


358 


343 


701 


51.07 


Mismanages instructional time 


(T2) 


527 


174 


701 


75.18 






15,103 


5,228 


20.329 


74.29 



1 



Table 6. reports the percentage of participants marking within the 70% 
range when broken down by tape pre and post. Note that the percent correct 
decreased in five out of six tapes from pretest to posttest. Tape 5 had 73.1% correct in 
the pretest and improved one percent to 74.1% in the posttest The results from pre to 
posttest resulted in a loss ranging from .9% on ♦ape 6 to 4.3% on tape 3. Ordinarily, 
the retest percentage correct would tend to decrease slightly over time as was the 
case in five of the six tapes. The markings remained stable over the time between pre 
and posttesting. 



TABLES. 

XumbcT and Percentage Correct for Pre- and Posl-tesl by Tape 

Hashmark Section 



Pre Post 



Tape 


TOT 


TOT 


TOT 


% 


TOT 


TOT 


TOT 


% 


Number 


IN 


OUT 


TOT 




IN 


OUT 


TOT 




1 


1,360 


670 


2,030 


67,0 


1,124 


690 


1,914 


63.9 


2 


1,576 


425 


2,001 


78.8 


1,463 


451 


1,914 


76.4 


3 


1,078 


285 


1,363 


79.1 


889 


300 


1,189 


74.8 


4 


879 


339 


1,218 


72.2 


749 


353 


1,102 


68.0 


5 


1,421 


522 


1.943 


73.1 


1,376 


480 


1,856 


74 1 


6 


1,588 


355 


1,943 


81,7 


1,500 


356 


1.856 


80.8 



Table 7. presents the number and percent of subjects marking the checklist 
on the back of the KTlF classrrom observation instrument. To be included in the 
**W1THIN RANGE*' group, eight of the eleven items had to have been marked 
correctly. This criteria resulted in 89.23% of the markings being within the range on 
the pretest and 84.12% within range on the posttest. The least correctly marked tape 
on both the pretest and posttest was Tape 4. The resultant percentages indicate that 
tapes 1 through 5 had a higher percentage correct on the pretest than on the posttest. 
The largest loss was on Tape 5 which eroded 14.62 percent. Tape 6 markings resulted 
in a higher percent correct on the posttest than on the pretest. Overall some net 
reduction in correctness was expected with a SO day interval between the two data 
collections. The chi squared analysis comparing the proportions of respondents by in 
or out of range cate^ries with the expected ranges from the pretest resulted in a 
value of 6.43 which is not significant at the .20 level. 
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TABLK7. 

Number and Percent of Raters Meeting the Criteria for Teaching Behaviors 
(Checklist on Back of Coding Form) 

Pre Post 

Tape Within Out of Total Percentage Within Out of Total Percentage 
Number Range Range Marking Accuracy in Range Range Marking Accuracy m 

Range Range 



1 


61 


9 


70 


87.14 


54 


12 


66 


81.82 


2 


66 


3 


69 


95.65 


61 


5 


66 


92.42 


3 


43 


4 


47 


91.49 


38 


4 


42 


90.48 


4 


33 


9 


42 


78,57 


25 


13 


38 


65.79 


5 


59 


8 


67 


88.06 


47 


17 


64 


73.44 


6 


61 


6 


67 


91.04 


61 


3 


64 


95.13 


rotal 


323 


39 


362 


89.23 


286 


54 


340 


84.12 



In an effort to aid in producing a revised instrument and coding manual, 
the participants agreed to record comment and suggestions. The comments followed 
a consistent pattern from site to site, which was remarkable, because the presenters 
used various formats to present the orientation of the materials and the response. 
Some allowed participants to record comments in no particular format; others used a 
form developed by Kentucky State University (Appendix C) or the form developed by 
Murray State University (Appendix D). 



The comments most frequently made dealt with the location of a specific 
piece of information. The next most referenced commentary involved the suggested 
inclusion of another category to mark or exclusion of one or more of the current cate- 
gories. The items which were new, such as the content areas and the coding manual 
definition of how to mark, received some commentary, along with the inconsistent 
marking patterns. A few people commented on a philosophical point or basis which 
was previously discussed but not documented thoroughly in the coding manual. 

The pre-experimental design established that the instrument had sufii- 
cient reliability and validity to be used in the first year of the Kentucky program. 
The data collected indicated the need for a very sm ill group of knowledgeable people 
to review comments and concerns, to revise the coding manual, and to perform minor 
revisions on the coding instrument. These suggested revisions could still be accom- 
plished within the timeline of the overall process. Appendix E is the final copy of the 
observation instrument. 
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Kentucky Beginning Teacher Internship Program 
Classroom Observation Instrument 



Intmn's Name 
(Prli-.t) 

Intern's SS# . 

School Name _ 



Last 



First 

Date of Obsen/ation . 



Middte 



School Numbec 



District Name 
Starting Time. 



1 . Check method(s) used in observed lesson. 



District Number. 
Ending Time 



1. Lecture 

2. Discussion, Recitation. 

interaction 
3. independent Activity 

2 . Check the subject area observed. 

1 . Language Arts/English 

2. Mathematics 

— 3. Sciencd 

4 . Social Studies 

— 5. Music 
6. Art 



4. Combination of 1 and 2 

5. Combination of 1 and 3 

6. Combination of 2 and 3 

7. Combination of all 



. 7. Physical Education 

. 8. Industrial Education (5-12) 

. 9. Home Economics Education (5-12) 

• 10. Vocatkjnal Education 

■ 11. Special Educatton 

.12. Other 



3 . Check type of classroom in which observation occurred. 



1 . Self-contained Classroom 

2. Field or Court 

3. Library/Media Center 



4. Laboratory/Shop 

5. Resource Room 

6. Other 



Provide the following information for the class observed. 



4. 

5. 
6. 
7. 



Grade level(s) of students 

Total number of students in class 

Number of students having a lEP 

Intern's number of completed years teaching experience 



Length of coding (in minutes). 

Ooserver's name (print) 

Observer's signature 

Intern's signature 
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LESSON COMPONENTS 



TEACHING BEHAVIORS 




OBSERVATION NOTES: 



Conient 




1. 5ial6&'^Ucni ICS LLiiilt;ni lUvUd 


































2. exDlams content ciearly 


































3. models/aoDlies content focus 


































4. checks for comorehension 


































5. provides Quided Dracuce 


































6. conducts review/summary 


























1 






7. Dfovides moeoendpnt practice 





































1 asks acaoemic Questions 








2. asKs muitiDie questions 




c 




3. asks/oauses/narr.es 




JZ 

u 




4. does not provide wait time 




c 




5. quides reate?- 




c 
g 




6. does not guide reciter 




07 
O 




7. allows catt-outs 






8. asks procedural questions 








9. asks unrelated questions 








1. acknowledges student responses 




in 




2 reohrases/amolifies student responses 




u> 
c 




3. corrects/ciar»ttes student responses 




a: 




4. does not correct/clanfy responses 








5. resDoncL to academic auestions/input 








0. reSDOnCS IC nondCdatrnnC UUt?ol'Uilb/inuui 








1, us:5 soecific academic praise 




</> 
S 
a 




2 uses General academic praise 






3. uses group acadennic praise 








4, uses conduct-related praise 




g 




1 . cues students 




m 
u 
c 




2. uses emphasis 




6 




3. uses challenqe 




E 
o 
o 




4. uses sarcasm/negative affect 




c 




1. stops misconduct positively 








2. stops misconduct negatively 




o> 
re 
c 




3, does not stop misconduct 








4. manages overlapping events 




is 




1. minimizes management time 






2. mismanages instructional time 


"^Hl 
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DURING THIS OBSERVATION, THE INTERN: 

I began instruction promptly. 

2 secured the attention of students. 

3. reviewed students' previous work or previous lesson content. 

4 provided an overview of lesson content or a statement of lesson objective{s). 

5 proviOea an overview ot now the iesson would proceed. 

fi elicited responses from a variety of students in a nonrepetitive pattern qi 

used ordered turns (only in K-3 math & reading). 

7. used appropriate vocabulary/correct grammar. 

ft provided clear direction for seatwork/homework {write NA if not applicable). 

9 circulated and monitored students' progress during seatwork (write NA if not 

applicable). 

10. was involved. 

I I was enthusiastic. 
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SUMMARY OF PARTICIPANT DATA FOR THE AUGUST, 1988 
PILOT PHASE OF THE TEACHER INTERN PROJECT 



TAPE CODED 









Cumulative 


Cumulative 




Frequency 








1 


69 


19.1 


69 


19.1 


2 


68 


18.8 


137 


37.8 


3 


47 




1 AA 


ft 

Qw • O 


4 


42 


X J. • D 






5 


68 


18.8 


294 


81.2 


6 


68 


in p 
1 o . o 






ASSIGNED TEACHING LEVEL 














Cumulative 


Cumulative 




Frequency 


Percent 


Frequency 


Percent 


ELEMENTARY 


139 


oo . 4 


1 QQ 


oc5 • 4 


MIDDLE 


89 


A C 

Z4 . O 


O O Q 


AO n 
Do • U 


HIGH 


134 


O f .\J 




1 f\r\ f\ 
XUU • U 




ATTENDED 














wuntU 1 o u 1 V e 


C'Umu X o X. X V 6 




Frequency 


Percent 


Frequency 


Percent 


NORTHERN KY 


52 


14.4 






EASTERN KY 


3& 


9.9 


88 


24.3 


MOREHEAD ST 


56 


15.5 


144 


39.8 


KY STATE 


58 


16.0 




iJ ^ . o 


U OF L 


50 


13.8 




gq 6 


MURRAY ST 


54 


14.9 






U OF K 


56 


15.5 


*>J^ vJ ^ 


1 on n 


OCCUPATION 


OF PARTICIPANT 












Cumulative 


Cumulative 




Frequency 


Percent 


Frequency 


Percent 


PRINCIPAL 


79 


21.8 


79 


21.8 


RESOURCE 


85 


23.5 


164 


45.3 


SUPERVISOR 


80 


22.1 


244 


67.4 


COLLEGE 


82 


22.7 


326 


90.1 


EXCEPT ED 


14 


3.9 


340 


93.9 


VOC ED 


22 


6.1 


362 


100.0 
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KENTUCKY BEGINNING TEACHER INTERNSHIP PROGRAM 

PILOT PROJECT 

Clifton L. McMahon, Ph.D., Regional Coordinator 
EVALUATION OF CLASSROOM OBSERVATION INSTRUMENT 

This form is to be completed immediately after the 
observation and brought to the meeting on November 19. 

I. General Information Section: 

1. Does the form call for all needed information? 

2 . Is unnecessary information requested? 

3. Are all the items and categories clear and adequate? 

4 . Other comments? 

II. Teaching Behaviors: 

1. Are the seven areas of behavior adequate? 

2. What difficulties were encountered in coding and 
tallying the intern's performance? 

(CONTINUED ON BACK) 



3 



Should adjustments (additions-deletions) be made in the 
items in any of the seven areas? 



4. Was the section for tallying seat work useful? 

5. Other comments? 

III. Post-Observation Check Section: 

1. Did you encounter problems in recalling behavior in 
order to complete this section? 

2. Is there an advantage to checking this section after 
the conclusion of the lesson? 

3. What adjustments should be made to this section? 

4 . Other comments? 
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Kentucky BeQinnlng Ttachtr Inttrnshlp Program 
PILOT RESEARCH PROJECT QUESTIONNAIRE 
Novtmber 1^, 1988 



1. How wen prtpared were you <training-wist> to code this instrument in the -field? 



2. How dH4icult do you think it will be to train the teachers and administrators in 
our service region to reliably code with this instrument? 



3. Speci-fical ly. . . what is the minimum number o-f days which will be required to train 
administrators and teachers in our region to reliably code with this instrument? 



4, Is this instrument easier or harder to use in the -field than the FPMS instrument? 



5, Specif i call y. . . In what ways is this instrument easier to use than the FPMS? 



6. Specl-f Ically... in what ways is this instrument harder to use than the FPMS? 



7. What reconwendations do you make -for the use o-f the subject matter section which 
will make it easier/better to use. 



6. What do you think will be the attitude o-f your colleagues towards this instrument 
once they have been trained with it and start to use it? 



o r. 
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Kentucky Teacher Internship Program 
Classroom Observation Instrument 



Intern's Name. 
(Print)" 



Intern's SS# . 
School Name 
District Name 
Starting Time 



Last 



First 

Date of Observatton 
Sctx>ol Nun^r 
District Number 
Ending T\me 



Middle 



1 . Check the subject area observed. 

1. Language Arts/English 

2. Mathematics 

3. Science 

4. Social Studies 

^—5. Music 
6. Art 



.7. Phystoai Education 

.8. Industrial Education (5-12) 

.9. Home B»nomics Ec&icatton (5-12) 

.10. Vocational EdiKJatlon 

.11. Special Educatten 

. 1 2. Other 



2. Chedt type of classroom in which observatton occurred. 



.1 . Self-contained Classroom 

.2. Field or Court 

•3. Libraiy/Media Center 



.4. Laboratory/Shop 
■5. Resource Room 
.6. Other 



Provide the following infonnatlon for the class observed. 

3. Grade level(s) of students 

4. Total number of students in class 

5. Number of students having an lEP 



6.. 



Intern's number of completed years teaching experience 
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Length of coding (in minutes), 



Observer's name (print) 



Observer's signature 
Intern's signature — 
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OBSERVATION NOTES: 



LESSON COMPONENTS 



TEACHER BEHAVIORS 








1 . states/defines lesson component 






























c 




2. explains component clearly 






























s 


d 


3. demonstrates component 






























1 




4. provides guided practice 


































5. provides independent practice 































TEACHER BEHAVIORS 



DURING INSTRUCTION 



c 
o 




1. cues students 






SJ 

C 




2. uses emphasis 






E 




3. uses challenge/task attractions 






E - 
o 




4. uses sarcasm/negative affect 










1. asks academic questions 










2. asks multiple questions 






cr - 
*c 

X 




3. allows catl-outs 






y 




4. does not provide wait tim.: 






c 
c 




5. guides reciter 







0 




6. does net guide reciter 






a 




7. asks procedural questions 










8. asks unrelated questions 










1 . acknowledges student responses 


— — — 




S 
c 

§. 

IS 




2. rephrases /amplifies student responses 








3. corrects/clarifies student responses 










4. does not correct/clarify responses 










1 . uses specific academic praise 








2. uses general academic praise 




DURING SEATWORK 


a 




3. uses group academic praise 












1. uses specific conduct-related praise 








c 




i 2. stops misconduct positively 








E 




3. stops misconduct negatively 








c 




4. does not stop misconduct 








2 




5. manages overlapping events 












1. minimizes management time 








E 




2. mismanages instructional time 












J _ rrrr — ■ 

E-2 



♦ *0 
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Lesson Initiation 



1. 

lesson promptly 



began observed 



2. secured the 

attention of students 
before beginning instruct- 
ion. 



3._^ ^used a review of 

students' previous work or 
previous lesson content to 
establish academic focus. 



4. provided an over 

view ot lesson content 
or a statement of lesson 
objective(s) to establish 
academic focus. 



5. p rovided an over 

view ot how the lesson 
would proceed to establish 
academic focus. 



TEACHER BEHAVIORS TO BE CHECKED AT END OF LESSON 
DURING THIS OBSERVATION. THE INTERN: 



1. asked questions, paused, then identified respondents. 

2. elicited responses from a variety of students in a nonrepetitive pattern to check 

for understanding or 

elicited responses from students in ordered turns to check for understanding 
(appropriate only in K-3 small group instruction in math and reading). 

3. used appropriate grammar and vocabulary. 

4. conducted a review or summary of major lesson components. 

5. provided clear directions for seatwork/homework. (Enter NA if not applicable.) 

6. circulated and monitored students' progress during seatwork. (Enter NA if not 

applicable.) 

7. was involved. 

8. was enthusiastic. 



OBSERVER'S SUMMARY STATEMENT (OPTIONAL): 
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AFTER COMPLETING AN OBSERVATION. PLEASE CHECK TO INSURE THAT: 

1 ALL INFORMATION ON THE FRONT PAGE OF THE INSTRUMENT IS COMPLETE 



2 EACH LESSON COMPONENT HAS A CHECK (/oR ZERO (0) ENTERED f OR EACH TREATMENT BEHAVIOR 

3 THE NUMBER OF TALLIES HAS BEEN t NIEREO IN THE BOX TO THE LEFT OF EACH BEHAVIOR 

4 EACH LESSON INlTIAUON AND END OF LESSON BEHAVIOR HAS BEEN MARKED WITH A CHECK {/zERO <0). OR NA 
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